Proceedings of the Fifth Dutch - Belgian Information Retrieval

Todays content is increasingly a mixture of text, multimedia, and metadata. One way to format this mixed content is according to the adopted W3C standard for information repositories, the so-called eXtensible Markup Language (XML). The increasing use of XML in scientific data repositories, Digital Libraries and on the Web, has brought about an explosion in the development of XML tools, and in particular systems to store and access XML content. Whereas many of todays access tools still treat documents as single large (text) blocks, XML offers the opportunity to exploit the internal structure of documents in order to allow for more precise access thus providing more specific answers. Providing effective access to XML-based content is therefore a key issue. Providing effective access to XML-based content is what XML retrieval research is about. XML retrieval systems aim to exploit the logical structure of documents, which is explicitly represented by the XML markup, to retrieve document components (the so-called XML elements) instead of whole documents in response to a user’s query. Implementing this more focused retrieval paradigm means that an XML retrieval system needs not only to find relevant information in the XML documents, but also determine the appropriate level of component granularity to return to the user. It is this goal that makes the development AND the evaluation of XML retrieval approaches challenging. INEX, the Evaluation Initiative for XML Retrieval, is an initiative currently supported by DELOS; a network of excellence in digital libraries (http://www.delos.info/). INEX was set up at the beginning of 2002 with the aim to establish infrastructures, XML test-suites and appropriate measurements for evaluating XML retrieval approaches. INEX is responsible for a range of evaluation activities in the field of XML information access. INEX 2004 had five tasks, where each task was designed to test particular aspects of XML retrieval: ad-hoc retrieval, interactivity, natural language querying, relevance feedback, and heterogeneous content. A large number of institutions (The following institutions contributed to the organization of INEX 2004: U Duisburg-Essen (D), QMUL (UK), U Amsterdam (NL), LIP6 (F), U Otago (NZ), CWI (NL), QUT (AUS), IBM (IL), LIS (DK), U Minnesota-Duluth (US)) are strongly involved in the organization of these activities, by contributing to the evaluation methodologies and providing evaluation software and tools. After three years of INEX, we are still needing further research in order to arrive at the correct methodology for evaluating XML retrieval, in particular with respect to returning the appropriate level of component granularity. This talk will discuss the issues involved in returning the appropriate level of component granularity to the user in the context of INEX, and before INEX, as encountered in some of my previous work. My aim here is to both present the evaluation methodology followed by INEX, and to obtain feedback from the attendees as the issues we are dealing with concern many other areas. For more details about INEX 2004, see http://inex.is.informatik.uni-duisburg.de:2004/.